Apparatus and method for managing data storage

ABSTRACT

An apparatus for controlling a log-structured data storage system, operable with a first log-structured data storage area for storing data, comprises a metadata storage component for controlling the first log-structured data storage area and comprising a second log-structured data storage area for storing metadata; and means for nesting the second log-structured data storage area for storing metadata within the first log-structured data storage area. The apparatus may further comprise at least a third log-structured data storage area for storing further metadata, and means for nesting the at least a third log-structured data storage area within the second log-structured data storage area.

CROSS-REFERENCE TO RELATED FOREIGN APPLICATION

This application is a non-provisional application that claims prioritybenefits under Title 35, Unites States Code, Section 119(a)-(d) fromEuropean Patent Application entitled “APPARATUS AND METHOD FOR MANAGINGDATA STORAGE” by Henry Esmond BUTTERWORTH and Ian David JUDD, havingEuropean Patent Application No. EP07150140.7, filed on Dec. 19, 2007,which patent application is incorporated herein by reference in itsentirety.

FIELD OF THE INVENTION

The present invention relates to a technology for improved management ofdata storage, and more particularly to a technology for managing alog-structured data storage system, such as a log-structured file systemor array.

BACKGROUND OF THE INVENTION

In Rosenblum et al, “The Design and Implementation of a Log StructuredFile System,” Proceedings of the 13th ACM on Operating SystemPrinciples, October 1991, a log structured file system was proposedwhere modified data blocks are re-written to the disk sequentially in alog-like structure. Information (metadata) is also written with eachwrite operation about the data being written. This metadata is used inmanaging the system.

The concept of log-structured file systems and arrays (LSAs) is nowwell-known to those of ordinary skill in the art, and need not befurther described here. For the purposes of the present description, theterm “LSA” will be used throughout, for the sake of brevity, but it willbe understood by one of ordinary skill in the art that the term mayencompass any log-structured data storage arrangement, such as an arrayor a file system. An example of a conventional arrangement of storageapparatus incorporating an LSA is shown in FIG. 1, in which virtualaddress space 100 is mapped to storage under control of conventional LSA102. Conventional LSA 102 makes use of three data areas: superblock area104, track data area 106 and segment data area 108. The operation ofthese data areas will be explained further below.

One significant problem with the management of conventional LSA is theproblem of taking backups of LSA snapshots.

With a conventional LSA, it is possible to make snapshot copies ofextents of a customer-visible address space such that the same data isaccessible through multiple extents of the address space but only onephysical copy is stored. The ability to make these snapshots hasnumerous benefits described in the literature. One such benefit is theability to make a snapshot of a running system at a particular point intime and store that to backup media as a consistent point in time recordof the state of the system whilst allowing the system to run on whilstthe backup is in progress. Another benefit of snapshot operations is theability to make very many copies of some information and allow multipleclients access to one copy each but only store one physical copy of thedata plus any changes the clients make to their own copies. An exampleof the latter use of snapshot might be a service provider providingremote storage for a stateless personal computing device: the serviceprovider could provide each new client with a snapshot of a standardvolume and the client could modify its own volume as needed. The serviceprovider would need far less storage with this approach than if it wereto allocate physical storage for each client volume up front.

The problem with conventional LSA in this area is that it is notpossible to make use of both of those features and take a point in timebackup of data which itself contains multiple snapshot extents. In fact,taking the point-in time backup works fine but when it comes torestoring the data the information about snapshots in the backup extentis not preserved and the restored data will no longer fit on thephysical storage.

A second significant problem not addressed by conventional LSAtechniques is that of scalability.

With a conventional LSA, there is a two-tier LSA directory whichcontains metadata that maps the virtual address space seen by the clientapplication to the physical address space presented by the underlyingphysical storage. In order to preserve the LSA metadata across poweroutages it is necessary to allocate space for it in non-volatile storagesomewhere. Typically it is stored on the underlying physical storagealong with the customer data.

There are valid uses of the LSA snapshot feature which require an LSAimplementation to present a vast address space to the customer and theproblem with a conventional two-tier directory is that in order topresent a vast address space, a significantly less but not negligibleamount of physical storage space must be allocated for the directory.This imposes scalability constraints because, for example, the minimumamount of physical storage the customer must buy must be at least largeenough to hold the meta data for the maximum virtual address space thecustomer might ever want. Whilst the factor between the size of themetadata and the virtual address space is quite large (say 1000×) thisis offset by the number of times the data is snapshot so for example ifdata was on average snapshot 1000× then the minimum configuration wouldhave to be at least half the size of the maximum configuration in orderto hold all of the metadata for the maximum configuration.

One possible approach to this problem is to allocate the metadatadynamically, for example in a b-tree, and grow the amount of physicalstorage allocated to metadata as the customer's requirement for virtualaddress space increases. This approach is valid but it introducessignificant extra implementation cost to the design of a fault tolerantLSA because of the complexity of manipulating the meta-meta-datastructures.

A third significant problem not addressed by conventional LSA techniquesis that of the performance of snapshot over very large address spaces.

It is important that snapshot operations happen as quickly as possiblebecause, when they are used for taking point-in-time backups, whilstthey are in progress the customer application is usually suspended andthis backup-window usually represents lost business for the customer.

Conventional LSA implementations with a two tier directory do areasonable job with snapshot but again have a scalability problembecause the time taken to perform a snapshot is proportional to the sizeof the virtual extent being snapshot whether or not that extent has beenwritten with data. This is because the directory must be scanned entryby entry to perform the snapshot whether or not the directory entrieshave ever been written.

It is reasonable to imagine a small storage service provider startingwith, for example, a few terabytes of physical storage with theintention of scaling to a few exabytes using a virtual address spacelarge enough for that amount of data and for a single snapshot of it andwishing to perform nightly backups of its clients' data by taking alarge snapshot at midnight and spooling it off. With a conventional LSA,there would be a few problems with this naive approach, the mostsignificant of which would be traversing all of the unused address spacewhich would take a long time. Of course, these problems can all beaddressed with the application of knowledge and experience by thestorage service provider in selecting appropriate extents to back up butthe requirement for an intelligent administrator translates into highercost of ownership and possibly lower availability (because of thepotential for human error).

A fourth significant problem not addressed by conventional LSA is theproblem of LSA metadata writes reducing LSA scalability and performance.

When data is written to an LSA, the metadata must be updated to reflectthe new location of the data. The conventional LSA approach is to writemetadata changes to a journal in fast non-volatile memory and to hardenthe journal periodically to a copy of the metadata stored at a fixedlocation on the underlying physical storage.

This approach has the problem that, when the virtual address space isvastly larger than the working set and the working set is sparselydistributed in that virtual address space, hardening the journal resultsin a random disk write for each customer data write and thereforeeliminates LSA's advantage of collating writes into segments in thefirst place. This limits the scalability and usability of conventionalLSA to applications which do not exhibit this kind of workload.

It would thus be desirable to have an improved technology for managingdata storage, and more particularly for managing a log-structured array(LSA) storage system.

SUMMARY

The present invention accordingly provides, in a first aspect, anapparatus for controlling a log-structured data storage system, operablewith a first log-structured data storage area for storing data, andcomprising: a metadata storage component for controlling said firstlog-structured data storage area and comprising a second log-structureddata storage area for storing metadata; and means for nesting saidsecond log-structured data storage area for storing metadata within saidfirst log-structured data storage area.

The apparatus may further comprise at least a third log-structured datastorage area for storing further metadata, and means for nesting said atleast a third log-structured data storage area within said secondlog-structured data storage area.

Preferably, unused metadata is recognisable patterned data, and saidsecond log-structured data storage area for storing metadata isallocated on demand.

Preferably, the unused metadata comprises an all-zeroes pattern, or theunused metadata comprises an all-ones pattern.

The apparatus may further comprise a compression component forcompressing said metadata before writing.

The apparatus may further comprise a snapshot component for taking asnapshot copy of said data.

Preferably, said snapshot component comprises: a divider sub-componentfor dividing each log-structured data storage area for storing metadatainto a working half and a backup half, a first copier sub-component forcopying said metadata from said working half to said backup half of eachlog-structured data storage area for storing metadata; a determinersub-component for determining for a record in said backup half of eachlog-structured data storage area for storing metadata whethercorresponding data in a log-structured data area is to be copied; and asecond copier sub-component, responsive to said determining giving apositive result for said record, for copying said corresponding data.

The apparatus may further comprise a data deduplication componentcomprising: a hashing component for creating and storing a hash value ofa data entity; a comparator for comparing hash values of a pair of dataentities; a write control component responsive to an output of saidcomparator for selectively writing a non-duplicate data entity and forcreating a snapshot of a duplicate data entity.

The apparatus may further comprise a write cache component, and saiddata deduplication component is operable during a cache destage.

In a second aspect, there is provided a method for controlling alog-structured data storage system, operable with a first log-structureddata storage area for storing data, and comprising the steps of:controlling, by a metadata storage component, said first log-structureddata storage area and providing a second log-structured data storagearea for storing metadata; and nesting said second log-structured datastorage area for storing metadata within said first log-structured datastorage area.

The method may further comprise providing at least a thirdlog-structured data storage area for storing further metadata, andnesting said at least a third log-structured data storage area withinsaid second log-structured data storage area.

Preferably, unused metadata is recognisable patterned data, andcomprising the further step of allocating said second log-structureddata storage area for storing metadata on demand.

Preferably, the unused metadata comprises an all-zeroes pattern, orwherein the unused metadata comprises an all-ones pattern.

The method may further comprise the step of compressing said metadatabefore writing.

The method may further comprise taking a snapshot copy of said data.

Preferably, said taking a snapshot comprises: dividing eachlog-structured data storage area for storing metadata into a workinghalf and a backup half, copying said metadata from said working half tosaid backup half of each log-structured data storage area for storingmetadata; determining for a record in said backup half of eachlog-structured data storage area for storing metadata whethercorresponding data in a log-structured data area is to be copied; andresponsive to said determining giving a positive result for said record,copying said corresponding data.

The method may further perform steps of data deduplication comprising:creating and storing a hash value of a data entity; comparing hashvalues of a pair of data entities; responsive to an output of saidcomparator, selectively writing a non-duplicate data entity and creatinga snapshot of a duplicate data entity.

The method may further comprise performing write caching, wherein saiddata deduplication is performed during a cache destage.

In a third aspect, there is provided a data carrier having functionaldata thereon, the functional data comprising functional computer datastructures to, when loaded into a computer system and operated uponthereby, enable the computer system to perform all the steps of a methodaccording to the second aspect.

In a fourth aspect, there is provided a computer program comprisingcomputer program code to, when loaded into a computer system andexecuted thereon, cause the computer system to perform all the steps ofa method according to the second aspect.

The preferred embodiments of the present invention thus create a newkind of LSA by nesting a conventional LSA implementation inside itselfiteratively and using the LSA features such as snapshot, compression andpacking writes into segments of the inner LSA instances to improve thecharacteristics of the new LSA as a whole. The advantages includeimprovements in backup, scalability, snapshot performance and in theefficiency of metadata writes.

The preferred embodiments of the present invention provide methods ofimplementing a Log Structured Array (or file system) using a stack ofLSA-like subcomponents which have all of the benefits of LSA asdescribed in the literature and a number of additional benefits.

Iterative LSA allows snapshot backups of extents containing snapshotinformation whilst preserving that snapshot information and provides theguarantee that a backup of an extent containing multiple snapshots willfit back on the physical storage when it is restored.

With iterative LSA, allocation of physical storage for metadata happensnaturally on demand with very little additional design or coding effortover the conventional upfront allocation approach and it is possible tostore all the metadata for a vast address space in virtually no physicalstorage and grow the storage subsystem from a very small amount ofstorage to a vast amount to meet the customer demand. As a concreteexample, it would be entirely reasonable to implement an iterative LSAwhich presented (say) a 64 bit address space using only a single (say)10 GB disk (or for fault tolerance, only a single array). This has thebenefit that the customer has a very low entry level cost and has thepotential to scale to systems as large as they are ever likely to need.

Iterative LSA is structured such that the performance of snapshotoperations scales with the logarithm of the size of the extent insteadof being proportional to it as for a conventional LSA. This means thatthere is little harm in snapshotting unused extents of the virtualaddress space for backups as those extents can be copied at virtually nocost and the naive strategy outlined above for backup is entirelyreasonable.

Iterative LSA addresses the problem of the performance impact ofmetadata writes for typically random workloads because metadata writesare collated into segments in the same way that customer data writes arein conventional LSA; thus the number of disk head movements is notincreased significantly by the requirement to harden the metadata.

Embodiments of the present invention, in its broadest aspect, provideapparatus and methods for managing data storage, and more particularlyfor managing a log-structured array (LSA) storage system.

BRIEF DESCRIPTION OF THE DRAWINGS

A preferred embodiment of the present invention will now be described,by way of example only, with reference to the accompanying drawingfigures, in which:

FIG. 1 shows in schematic form an LSA apparatus or arrangement accordingto the prior art;

FIG. 2 shows in schematic form one possible apparatus or arrangementaccording to one possible improvement to the prior art;

FIG. 3 shows in simplified schematic form an LSA apparatus orarrangement according to a preferred embodiment of the presentinvention;

FIG. 4 shows in simplified schematic form an LSA apparatus orarrangement according to one possible further modification of apreferred embodiment of the present invention;

FIG. 5 shows in simplified schematic form an arrangement of componentsaccording to one embodiment of the present invention; and

FIG. 6 shows in flowchart form, the steps of a method of taking apoint-in-time backup copy according to a preferred embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A preferred embodiment of the present invention will now be described,with reference to the figures. As described above, FIG. 1 depicts anarrangement or apparatus according to the prior art. This has beendescribed above as forming the background to the present invention.

A conventional LSA as shown in FIG. 1 has underlying physical storagefor some kind of superblock (call this the superblock area 104), for itsmetadata stored in metadata tracks (call this the track data area 106)and for the customer data stored in segments (call this the segment dataarea 108). Using this underlying physical storage, the LSA provides avery large virtual address space 100 which can be sparsely populated andwhich supports snapshot operations.

Now, the scalability problem outlined above (the “second significantproblem”) is that to support a very large virtual address space, thetrack data area has to be quite large (about 1000th as big as thevirtual address space) so if a system is needed which will scale to areally large virtual address space then it is necessary to start with avery large track data area and if it is necessary to allocate it upfrontthen the customer has to buy storage to hold all the data. So thesolution is not to allocate the track data area upfront but to allocateextents of it when they are required. This means that the track dataarea is not simply written to a contiguous range of sectors on disk butis stored in a data structure which will support sparse allocation ondemand.

One approach to this problem is to use a b-tree, as shown in FIG. 2. InFIG. 2, a b-tree is used to hold the track data area 210 so the trackdata area can be allocated on demand and thereby implement systems whichcan scale from very small amounts of physical storage yet still presentvery large virtual address spaces. The concept of b-trees is well knownto those of ordinary skill in the art.

The problem with implementing the b-tree is that to do it in a faulttolerant way requires considerable design and coding effort to add tothe already significant effort of implementing the conventional LSA.

In the preferred embodiment of the present invention, therefore, theb-tree solution is not adopted. Instead, the concept of an “iterativeLSA” or “nested LSA” is introduced, whereby a first LSA is implementedas normal, but its control data or metadata is, in turn, stored in asecond LSA nested within the first. This arrangement is shown in FIG. 3.One advantage of such an arrangement, from the point of view of animplementer of such a system, is that the essential logic arrangementsor corresponding code arrangements for an LSA have already been createdfor the first LSA, and can thus be reused for the second LSA.

Thus, in FIG. 3 are shown two instances of an LSA 302A and 302B whichshare the superblock area 304 and the segment data area 308. The inner(and smaller) instance of the LSA 302B uses its share of the superblockarea 304B and segment data area 308B and a small track data area 306B toprovide a virtual track-data area in which the larger LSA instance canstore its metadata. Now, provided the LSA is coded such that themetadata is all zeroes when it is not in use, the upper LSA 306A canpresent a large virtual address space and the lower LSA 306B will onlyallocate physical storage for the metadata for that address space whenthe upper LSA actually needs it. The lower LSA 306B level can allocateits metadata upfront in a simple way.

Now, as the track data area of the lower LSA instance 306B is about1000× smaller than the track data area for the upper instance, there isa saving of about a factor of 1000 in the amount of storage space if thetrack data area of the lower instance is allocated upfront. 406 b

Further advantageously, it is possible to iterate to as many levels asnecessary and, with the addition of each level, the amount of upfrontstorage allocation by the lowest level shrinks by a factor of about1000.

So, it would be possible to have three levels of LSA, as shown in FIG.4.

In FIG. 4, there are shown three LSA instances 402A, 402B, 402C sharingthe superblock area 404 and the segment data area 408. The lowest LSAinstance 402C has a very small upfront allocated track data area 406Cwhich it uses along with its share of the other areas 404C and 408C, topresent a larger virtual address space in which the LSA instance 402Bdirectly above stores its metadata, and so on for each further level.

With enough levels, it is possible to present a very large address spacewith only a very small upfront allocation of storage for the lowestlevel.

Aside from actually implementing a conventional LSA, there is onedifficulty with this iterative or nested LSA architecture: that ofdynamically sharing the segment data area 308 and 408 (FIGS. 3 and 4)between the LSA instances without deadlock. This can be addressed usinga storage reservation strategy, as the superblock area 304 and 404 issufficiently small that it can be allocated without the need for dynamicsharing between the LSA instances.

FIG. 5 shows in simplified schematic form a set of components accordingto a preferred embodiment of the present invention. Log-structured datastorage control apparatus 500 comprises a metadata storage component502, and a compression component 504. Apparatus 500 further comprises asnapshot component 506, which in turn comprises four sub-components: adivider 508, a determiner 510, a data copier 512 and a metadata copier514. Apparatus 500 further comprises a data deduplication component 516,which in turn comprises three sub-components: hasher 518, comparator 520and write control sub-component 522. Data deduplication component 516 isoperable in communication with write cache control component 524.

The fourth problem described as part of the background discussion abovewas that of the performance impact of performing metadata writes. Aswill be clear to one of ordinary skill in the art, once the metadataitself is stored in an LSA it is automatically collated into segmentsand is written out with fewer disk head movements, and thus theperformance problem is alleviated. In a further improvement, themetadata could also be compressed which should further improveperformance.

The first problem described in the Background section above was abouttrying to take snapshot backups which preserved snapshot information andcould be restored onto the same amount of storage as they came from In apreferred embodiment of the above-described iterative LSA architecture,this problem may be addressed by reserving half of the track data areaaddress space of each level for backup state (this reduces the 1000factor described above to 500 and might require the addition of anotherlevel to the multi-level LSA arrangement). Then, using the snapshotfunctionality of each level, it is possible to make a t0 (orpoint-in-time backup) which preserves snapshot relationships as follows:

-   -   1) Customer data I/O is quiesced and any cached data flushed to        the LSA.    -   2) The working half of each track data area is snapshot copied        to the backup half starting with the track data area at the        lowest level and working all the way up to the virtual address        space containing the customer data. This process makes a        snapshot copy of the working LSA state at this time (each        snapshot modifies the working state of the level below but,        because the snapshots are from the bottom up, by the time it is        performed the original working state of that level has already        been captured).    -   3) Customer data I/O may be restarted. It will be possible to        arrange that backup snapshots are on large power of two        boundaries so the backup window should be sub-second after the        customer application is stopped and the cache flushed.    -   4) While the customer application runs on the working copy, a        process may traverse the backup state of the LSA extracting        records to append to the backup image. By examining the backup        copy of the metadata of the lowest level and working upwards as        necessary the backup process can return the most compact set of        records for describing the LSA state.    -   5) Records could be of three types: zero extent records indicate        that an extent contains only zeroes; snapshot records indicate        that an extent is a snapshot of an extent which has already been        fully described; track records indicate that a track has been        encountered for the first time and must be fully backed up.    -   6) The backup image consists of a series of these records as        they are emitted by the traversal process. The traversal process        might easily emit tracks which were already compressed so there        would be no need for decompression and subsequent recompression        before writing to the backup medium.    -   7) Restoration starts with a new LSA (which need not have the        same underlying storage configuration as long as there is at        least the same amount of storage as in the original LSA) and        proceeds by replaying the backup image. Zero records are        replayed by performing a discard extent operation (or whatever        operation the LSA has to free up physical storage) but would        only be required for incremental backups as a new LSA would        already be zeroed. Snapshot records are replayed by performing        the snapshot and track records by writing the track.

Incremental backups can be performed by preventing the traversal processfrom emitting records which are dated before the time of the last backup(date information might be stored as a sequence number in every segmentwhich would almost certainly be required for other reasons anyway).

FIG. 6 shows in flowchart form the steps of a method of taking apoint-in-time backup of data according to one embodiment of the presentinvention. At START step 600, the process begins, and at step 602,customer data I/O is quiesced and the cache is flushed. The working halfof the track data area is copied to the backup half at step 604. At step606, customer data I/O may be restarted. At step 608, a backgroundprocess traverses the backup state of the metadata to determine whichdata records need to be copied. At step 610, the correspondingpoint-in-time backup records are written. The process completes at ENDstep 612.

Extent backup can be performed by starting the traversal process at anoffset into the LSA and stopping it before reaching the end. Extentbackups performed this way will have the desired property that they willstand alone and not reference any data not contained in the backup evenif the original data in the LSA in the extent backed up containedsnapshots of data outside the backup extent.

Full and incremental backups of part of the LSA are guaranteed to fitback into the space that they previously occupied when restored providedthe original data did not contain snapshots of data outside the backupextent. Backups of data containing snapshots of data outside the backupextent will cause those snapshots to diverge on restoration which willrequire extra free space to be available for restoration to besuccessful.

Full and incremental backups of the entire LSA are guaranteed to fitback onto the LSA as in this case there is no possibility of snapshotswithin the backup extent referencing data outside the backup extent.

Snapshot of the track data area can essentially be decomposed into aduplication of one part of the metadata associated with it (the uppertier of the two-tier LSA directory) and incrementing the referencecounts in another part (the lower tier). It is not really necessary toknow the exact reference count of a track in an LSA; it is onlynecessary to know when the reference count drops to zero so that thetrack can be freed.

In a system of iterated LSA levels, duplication of metadata is the sameas asking the next lower LSA instance to carry out a snapshot operationand once this has been performed, the required reference countinformation can be obtained without incrementing the reference counts inthe lower tier of the directory but instead by taking into considerationthe reference count of the track belonging to the LSA level below whichcontains the upper tier of the directory for the current LSA level.

Thus a large snapshot can be propagated down the stack of LSA instancesand converted into a much smaller snapshot operation for a lower leveland very much less work needs to be done.

A snapshot can only be propagated down a level if it covers an extentlarge enough to cover a whole track worth of metadata and is correctlyaligned otherwise it must be performed like a conventional LSA snapshot.This means that in general each level of the stack will perform a smallamount of conventional snapshotting for incorrectly aligned bits of thesnapshot at the beginning and end of the extent and pass the middleportion down to be performed more efficiently by the level below. Thisamounts to a logarithmic scaling of the amount of work with the size ofthe extent.

In a refinement of the iterative LSA of the preferred embodiment, it ispossible to implement deduplication for arbitrary non-zero data as wellas for all-zeros data.

This form of deduplication may be incorporated into the nested LSAscheme by reserving an extent of the underlying metadata address spacefor a b-tree of track hashes and using that to determine whether a trackis a duplicate, choosing to either implement a write for non-duplicatesor a track snapshot for duplicates when the track is written.

The nested LSA implementation of the preferred embodiment requires writecaching at each level for performance, and so the data deduplicationoperation can be performed off the critical path by doing it on cachedestage rather than when the write arrives.

It will be clear to one of ordinary skill in the art that all or part ofthe method of the preferred embodiments of the present invention maysuitably and usefully be embodied in a logic apparatus, or a pluralityof logic apparatus, comprising logic elements arranged to perform thesteps of the method and that such logic elements may comprise hardwarecomponents, firmware components or a combination thereof.

It will be equally clear to one of skill in the art that all or part ofa logic arrangement according to the preferred embodiments of thepresent invention may suitably be embodied in a logic apparatuscomprising logic elements to perform the steps of the method, and thatsuch logic elements may comprise components such as logic gates in, forexample a programmable logic array or application-specific integratedcircuit. Such a logic arrangement may further be embodied in enablingelements for temporarily or permanently establishing logic structures insuch an array or circuit using, for example, a virtual hardwaredescriptor language, which may be stored and transmitted using fixed ortransmittable carrier media.

It will be appreciated that the method and arrangement described abovemay also suitably be carried out fully or partially in software runningon one or more processors (not shown in the figures), and that thesoftware may be provided in the form of one or more computer programelements carried on any suitable data-carrier (also not shown in thefigures) such as a magnetic or optical disk or the like. Channels forthe transmission of data may likewise comprise storage media of alldescriptions as well as signal-carrying media, such as wired or wirelesssignal-carrying media.

A method is generally conceived to be a self-consistent sequence ofsteps leading to a desired result. These steps require physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical or magnetic signals capableof being stored, transferred, combined, compared, and otherwisemanipulated. It is convenient at times, principally for reasons ofcommon usage, to refer to these signals as bits, values, parameters,items, elements, objects, symbols, characters, terms, numbers, or thelike. It should be noted, however, that all of these terms and similarterms are to be associated with the appropriate physical quantities andare merely convenient labels applied to these quantities.

The present invention may further suitably be embodied as a computerprogram product for use with a computer system. Such an implementationmay comprise a series of computer-readable instructions either fixed ona tangible medium, such as a computer readable medium, for example,diskette, CD-ROM, ROM, or hard disk, or transmittable to a computersystem, via a modem or other interface device, over either a tangiblemedium, including but not limited to optical or analogue communicationslines, or intangibly using wireless techniques, including but notlimited to microwave, infrared or other transmission techniques. Theseries of computer readable instructions embodies all or part of thefunctionality previously described herein.

Those skilled in the art will appreciate that such computer readableinstructions can be written in a number of programming languages for usewith many computer architectures or operating systems. Further, suchinstructions may be stored using any memory technology, present orfuture, including but not limited to, semiconductor, magnetic, oroptical, or transmitted using any communications technology, present orfuture, including but not limited to optical, infrared, or microwave. Itis contemplated that such a computer program product may be distributedas a removable medium with accompanying printed or electronicdocumentation, for example, shrink-wrapped software, pre-loaded with acomputer system, for example, on a system ROM or fixed disk, ordistributed from a server or electronic bulletin board over a network,for example, the Internet or World Wide Web.

In one alternative, the preferred embodiment of the present inventionmay be realized in the form of a computer implemented method ofdeploying a service comprising steps of deploying computer program codeoperable to, when deployed into a computer infrastructure and executedthereon, cause said computer system to perform all the steps of themethod.

In a further alternative, the preferred embodiment of the presentinvention may be realized in the form of data carrier having functionaldata thereon, said functional data comprising functional computer datastructures to, when loaded into a computer system and operated uponthereby, enable said computer system to perform all the steps of themethod.

It will be clear to one skilled in the art that many improvements andmodifications can be made to the foregoing exemplary embodiment withoutdeparting from the scope of the present invention.

1. A system in communication with data storage, for storing data, andcomprising: a first log structured data storage area storing data in astorage device, wherein the first log structured data storage areaincludes a first portion of a segment data area in the storage device tostore user data; a first log structured data storage system operablewith the first log structured data area to store the data in the firstlog structured data area; a second log structured storage area in thestorage device storing metadata for the data in the first log structureddata storage area, wherein the second log structured data storage areafor storing the metadata is nested within the first log structured datastorage area, and wherein the second log structured data storage areaincludes a track data area and a second portion of the segment data areato provide a virtual track data area in which the second log structuredstorage system allocates the physical storage to store the metadata forthe data; and a second log structured storage system operable with thesecond log structured storage area, wherein the second log structuredstorage system allocates address space in the second log structuredstorage area for the metadata on demand when the first log structuredstorage system requires the allocation of address space for metadata fordata the first log structured storage system stored in the first logstructured storage area.
 2. The system of claim 1, further comprising athird log structured data storage area for storing further metadata ofthe second log structured data storage system, wherein the third logstructured data storage area is nested within the second log structureddata storage area.
 3. The system of claim 1, wherein unused metadata isrecognizable patterned data.
 4. The system of claim 3, wherein theunused metadata comprises an all-zeroes pattern or wherein the unusedmetadata comprises an all-ones pattern.
 5. The system of claim 1,further comprising a compression component for compressing the metadatabefore writing.
 6. The system of claim 1, further comprising a snapshotcomponent for taking a snapshot copy of the metadata in the secondstorage area.
 7. The system of claim 1, further comprising a datadeduplication component comprising: a hashing component for creating andstoring a hash value of a data entity; a comparator for comparing hashvalues of a pair of data entities; and a write control componentresponsive to an output of said comparator for selectively writing anon-duplicate data entity and for creating a snapshot of a duplicatedata entity.
 8. The system of claim 7, further comprising a write cachecomponent, and wherein said data deduplication component is operableduring a cache destage.
 9. The system of claim 1, wherein the storagedevice further includes a first portion of a superblock area used by thefirst log structured data storage system and a second portion of thesuperblock area used by the second log structured data storage system toprovide the virtual track data area.
 10. A system in communication withdata storage, for storing data, and comprising: a first log structureddata storage area storing data in a storage device; a first logstructured data storage system operable with the first log structureddata area to store the data in the first log structured data area; asecond log structured storage area in the storage device storingmetadata for the data in the first log structured data storage area,wherein the second log structured data storage area for storing themetadata is nested within the first log structured data storage area; asecond log structured storage system operable with the second logstructured storage area, wherein the second log structured storagesystem allocates address space in the second log structured storage areafor the metadata on demand when the first log structured storage systemrequires the allocation of address space for metadata for data the firstlog structured storage system stored in the first log structured storagearea; a divider sub-component to divide each log structured data storagearea for storing metadata into a working half and a backup half; a firstcopier sub-component to copy the metadata from the working half to thebackup half of each log structured data storage area for storingmetadata; a determiner sub-component to determine for a record in thebackup half of each log structured data storage area for storingmetadata whether corresponding data in a log structured data area is tobe copied; and a second copier sub-component to copy the data in the logstructured data area in response to determining that the data in the logstructured data area is to be copied.
 11. A method, comprising:providing a first log structured data storage area storing data in astorage device, wherein the first log structured data storage areaincludes a first portion of a segment data area in the storage device tostore user data; providing a first log structured data storage systemoperable with the first log structured data area to store the data inthe first log structured data area; providing a second log structuredstorage area in the storage device storing metadata for the data in thefirst log structured data storage area, wherein the second logstructured data storage area storing the metadata for the first logstructured data storage system is nested within the first log structureddata storage area, wherein the second log structured data storage areaincludes a track data area and a second portion of the segment data areato provide a virtual track data area in which the second log structuredstorage system allocates the physical storage to store the metadata forthe data; and providing a second log structured storage system operablewith the second log structured storage area, wherein the second logstructured storage system allocates address space in the second logstructured storage area for the metadata on demand when the first logstructured storage system requires the allocation of address space formetadata for data the first log structured storage system stored in thefirst log structured storage area.
 12. The method of claim 11, furthercomprising providing a third log structured data storage area forstoring further metadata of the second log structured data storagesystem, wherein the third log structured data storage area is nestedwithin the second log structured data storage area.
 13. The method ofclaim 11, wherein unused metadata is recognizable patterned data. 14.The method of claim 13, wherein the unused metadata comprises anall-zeroes pattern or wherein the unused metadata comprises an all-onespattern.
 15. The method of claim 11, further comprising compressing themetadata before writing.
 16. The method of claim 11, further comprisingtaking a snapshot copy of the metadata in the second storage area. 17.The method of claim 11, further comprising performing data deduplicationby performing: creating and storing a hash value of a data entity;comparing hash values of a pair of data entities to determine whetherthe data entity is a duplicate; and responsive to an output of comparingthe hash values, selectively writing a non-duplicate data entity andcreating a snapshot of a duplicate data entity.
 18. The method of claim17, further comprising performing write caching, and wherein the datadeduplication is performed during a cache destage.
 19. The computerreadable storage medium of claim 11, further comprising taking asnapshot copy of the metadata in the second storage area.
 20. The methodof claim 11, wherein the storage device further includes a first portionof a superblock area used by the first log structured data storagesystem and a second portion of the superblock area used by the secondlog structured data storage system to provide the virtual track dataarea.
 21. A method, comprising: providing a first log structured datastorage area storing data in a storage device; providing a first logstructured data storage system operable with the first log structureddata area to store the data in the first log structured data area;providing a second log structured storage area in the storage devicestoring metadata for the data in the first log structured data storagearea, wherein the second log structured data storage area storing themetadata for the first log structured data storage system is nestedwithin the first log structured data storage area; providing a secondlog structured storage system operable with the second log structuredstorage area, wherein the second log structured storage system allocatesaddress space in the second log structured storage area for the metadataon demand when the first log structured storage system requires theallocation of address space for metadata for data the first logstructured storage system stored in the first log structured storagearea; dividing each log structured data storage area for storingmetadata into a working half and a backup half; copying the metadatafrom the working half to the backup half of each log structured datastorage area for storing metadata; determining for a record in thebackup half of each log structured data storage area for storingmetadata whether corresponding data in a log structured data area is tobe copied; and copy the data in the log structured data area in responseto determining that the data in the log structured data area is to becopied.
 22. tangible computer readable storage device including acomputer program, that when loaded into a computer system and executedthereon, causes the computer system to communicate with a storage deviceand to perform operations, the operations comprising: providing a firstlog structured data storage area storing data in the storage device,wherein the first log structured data storage area includes a firstportion of a segment data area in the storage device to store user data;providing a first log structured data storage system operable with thefirst log structured data area to store the data in the first logstructured data area; providing a second log structured data storagearea storing metadata for the data in the first log structured datastorage area, wherein the second log structured data storage areastoring the metadata for the first log structured data storage system isnested within the first log structured data storage area, wherein thesecond log structured data storage area includes a track data area and asecond portion of the segment data area to provide a virtual track dataarea in which the second log structured storage system allocates thephysical storage to store the metadata for the data; and Providing asecond log structured storage system operable with the second logstructured storage area, wherein the second log structured storagesystem allocates address space in the second log structured storage areafor the metadata on demand when the first log structured storage systemrequires the allocation of address space for metadata for data the firstlog structured storage system stored in the first log structured storagearea.
 23. The computer storage readable medium of claim 22, wherein theoperations further comprise: providing a third log structured datastorage area for storing further metadata of the second log structureddata storage system, wherein the third log structured data storage areais nested within the second log structured data storage area.
 24. Thecomputer readable storage medium of claim 22, wherein unused metadata isrecognizable patterned data.
 25. The computer readable storage medium ofclaim 22, wherein the operations further comprise compressing themetadata before writing.
 26. The computer storage readable medium ofclaim 22, further comprising performing data deduplication byperforming: creating and storing a hash value of a data entity;comparing hash values of a pair of data entities to determine whetherthe data entity is a duplicate; and responsive to an output of comparingthe hash values, selectively writing a non-duplicate data entity andcreating a snapshot of a duplicate data entity.
 27. The computer storagereadable medium of claim 22, wherein the storage device further includesa first portion of a superblock area used by the first log structureddata storage system and a second portion of the superblock area used bythe second log structured data storage system to provide the virtualtrack data area.
 28. A tangible computer readable storage deviceincluding a computer program, that when loaded into a computer systemand executed thereon, causes the computer system to communicate with astorage device and to perform operations, the operations comprising:providing a first log structured data storage area storing data in thestorage device; providing a first log structured data storage systemoperable with the first log structured data area to store the data inthe first log structured data area; providing a second log structureddata storage area storing metadata for the data in the first logstructured data storage area, wherein the second log structured datastorage area storing the metadata for the first log structured datastorage system is nested within the first log structured data storagearea; providing a second log structured storage system operable with thesecond log structured storage area, wherein the second log structuredstorage system allocates address space in the second log structuredstorage area for the metadata on demand when the first log structuredstorage system requires the allocation of address space for metadata fordata the first log structured storage system stored in the first logstructured storage area; dividing each log structured data storage areafor storing metadata into a working half and a backup half; copying themetadata from the working half to the backup half of each log structureddata storage area for storing metadata; determining for a record in thebackup half of each log structured data storage area for storingmetadata whether corresponding data in a log structured data area is tobe copied; and copy the data in the log structured data area in responseto determining that the data in the log structured data area is to becopied.